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Abstract 



We investigate Higgs boson pair production at hadron colliders for Higgs 
boson masses rriH < 140 GeV and rare decay of one of the two Higgs bosons. 
While in the Standard Model the number of events is quite low at the LHC, 
a first, albeit not very precise, measurement of the Higgs self-coupling is 
possible in the gg — > HH — > bb'j^ channel. A luminosity-upgraded LHC 
could improve this measurement considerably. A 200 TeV VLHC could make 
a measurement of the Higgs self-coupling competitive with a next-generation 
linear collider. In the MSSM we find a significant region with observable 
Higgs pair production in the small tan (3 regime, where resonant production 
of two light Higgs bosons might be the only hint at the LHC of an MSSM 
Higgs sector. 
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I. INTRODUCTION 



The CERN Large Hadron Collider (LHC) is scheduled to begin operation in 2007, begin- 
ning a new era wherein the mechanism of electroweak symmetry breaking and fermion mass 
generation will be revealed and studied in great detail. Although alternative mechanisms ex- 
ist in theory, this is generally believed to be a light Higgs boson with mass itlh < 219 GeV [1]. 
More specifically, we expect a fundamental scalar sector which undergoes spontaneous sym- 
metry breaking as the result of a potential which acquires a nonzero vacuum expectation 
value. The LHC will easily find a light Standard Model (SM) Higgs boson with very moderate 
luminosity [2,3]. Moreover, the LHC will have significant capability to determine many of its 
properties [4,5], such as its fermionic and bosonic decay modes and couplings [6-9], including 
invisible decays [10] and possibly even rare decays to second generation fermions [11]. 1 

Starting from the requirement that the Higgs boson has to restore unitarity of weak 
boson scattering at high energies in the SM [13], perhaps the most important measurement 
after a Higgs boson discovery is of the Higgs potential itself, which requires measurement of 
the trilinear and quartic Higgs boson self-couplings. Only multiple Higgs boson production 
can probe these directly [14,15]. 

Recent literature is replete with self-coupling measurement studies. There are numerous 
quantitative sensitivity limit analyses of Higgs boson pair production in e + e _ collisions 
ranging from 500 GeV to 3 TeV center of mass energies [15-18]. For example, one neural 
net-based study concludes that a 500 GeV linear collider with an integrated luminosity of 
1 ab~ x [18] could measure the trilinear Higgs coupling A for m H = 120 GeV, where H — > bb 
decays dominate, at the 20% level. However, none of these analyses addressed the case of 
itlh > 140 GeV, where the Higgs boson mostly decays into W bosons. Studies exploring 
the potential of the LHC, a luminosity-upgraded LHC (SLHC) with roughly ten times the 
amount of data expected in the first run, and a Very Large Hadron Collider (VLHC), 
have come only very recently [19-22]. These studies investigated Higgs pair production via 
gluon fusion with subsequent decay to same-sign dileptons and three leptons via W bosons, 
and cover the broader range 115 < m# < 200 GeV. They established that future hadron 
machines can probe the Higgs potential for m H > 150 GeV. At the LHC, an integrated 
luminosity of 300 ftr 1 provides for exclusion of vanishing A at the 95% confidence level or 
better over the entire range 150 < m H < 200 GeV. A VLHC would provide for precision 
measurement over much of this mass range, similar to or better than the limits achievable at a 
3 TeV e + e~ collider with 5 ab _1 [17]. However, we previously concluded that hadron colliders 
could not probe the mass region itlh < 140 GeV sufficiently well to be meaningful [22]. 

We reexamine that conclusion in this paper, utilizing rare decay modes in Higgs boson 
pair production for m# < 140 GeV at future hadron colliders. We first review the definition 
of the Higgs boson self-couplings and briefly discuss SM and non-SM predictions for these 
parameters in Sec. II. An overview of the rare Higgs decay modes in the SM (predominantly 



An e + e~ linear collider with a center of mass energy of 350 GeV or more can significantly 
improve these preliminary measurements, in some cases by an order of magnitude in precision, if 
an integrated luminosity of 500 fb _1 can be achieved [12]. 
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6&77 final states) and our analyses of these channels appears in Sec. III. We consider the 
LHC, SLHC and a VLHC, which we assume to be a pp collider operating at 200 TeV with a 
luminosity of £ = 2 x 10 34 cm _2 s _1 [23]. In Sec. IV we establish the prospects of observing 
a pair of minimal supersymmetric Standard Model (MSSM) Higgs bosons in the 6677 and 
bbfi + fi" decay channels. We present our conclusions in Sec. V. 

II. HIGGS BOSON SELF-COUPLINGS 

The trilinear and quartic Higgs boson couplings A and A are defined through the potential 

V(vh) = \mW H + \vrf H + ^Xrjjj, (1) 

where rjn is the physical Higgs field, v = (V^Gf)' 1 ^ 2 is the vacuum expectation value, and 
Gf is the Fermi constant. In the SM the self couplings are 

~ 771 ^ 

A = A = X SM =2^- ( 2 ) 

Regarding the SM as an effective theory, the Higgs boson self-couplings A and A are per 
se free parameters, and S-matrix unitarity constrains A to A < 87r/3 [13]. Since future 
collider experiments likely cannot probe A, we concentrate on the trilinear coupling A in the 
following. The quartic Higgs coupling does not affect the Higgs pair production processes 
we consider. 

In the SM, radiative corrections decrease A by 4 - 11% for 120 < m H < 200 GeV [24]. 
Larger deviations are possible in scenarios beyond the SM. For example, in two Higgs doublet 
models where the lightest Higgs boson is forced to have SM like couplings to vector bosons, 
quantum corrections may increase the trilinear Higgs boson coupling by up to 100% [24]. 
In the MSSM, loop corrections modify the self-coupling of the lightest Higgs boson in the 
decoupling limit, which has SM-like couplings, by up to 8% for light stop squarks [25]. 
Anomalous Higgs boson self-couplings also appear in various other scenarios beyond the SM, 
such as models with a composite Higgs boson [26], or in Little Higgs models [27]. In many 
cases, the anomalous Higgs boson self-couplings can be parameterized in terms of higher 
dimensional operators which are induced by integrating out heavy degrees of freedom. A 
systematic analysis of Higgs boson self-couplings in a higher dimensional operator approach 
can be found in Ref . [28] . 

III. ANALYSIS 

At LHC energies, inclusive Higgs boson pair production is dominated by gluon fusion [29]. 
Other processes, such as weak boson fusion, qq — > qqHH [30], associated production with 
heavy gauge bosons, qq — > WHH, ZHH [31], or associated production with top quark pairs, 
gg, qq — > tiHH [19], yield cross sections which are factors of 10-30 smaller than that for 
gg — > HH [29] . Since HH production at the LHC is generally rate limited, we consider only 
the gluon fusion process. 
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Figure 1. SM Higgs branching ratios relevant to our analysis of HH production. For W + W~ 
and ZZ, one of the gauge bosons is off-shell. 

Because the total gg — > HH cross section at both the LHC and VLHC is quite small, at 
most one Higgs boson undergoing rare decay will allow for a reasonable number of events 
to work with. We therefore consider only final states containing one 6-quark pair, which is 
the dominant SM Higgs boson decay mode for m# < 135 GeV, as shown in Fig. 1. Our 
previous study demonstrated that at both LHC and VLHC, 46 and bbr + T~ final states 
are overwhelmed by backgrounds [22]. While the backgrounds are more moderate for the 
r-channel, the observable part of this decay mode unfortunately has multiple additional 
small branching ratios, and the detectors have rather low efficiency to identify the r-leptons. 
As charm quarks are even more difficult to tag than 6-quarks, and the QCD backgrounds 
become much larger due to similarly less fake-tag rejection, we can immediately discount 
any colored final states for the rare decay. Weak boson pairs certainly qualify as rare decays 
in this mass region, but cannot be used: the bbW*W and bbZ*Z — > bbi^i^vv final states 
suffer from a huge QCD top pair background. Similarly for pp — ^ HH — > bbZ*Z with one 
or more hadronically decaying Z bosons, and bbZ^y — > bbjjj, QCD processes with the same 
final states are likely to overwhelm the signal (here, W* and Z* denote off-shell W and Z 
bosons). The bbZ*Z — > bb + 4 leptons and — > £ + £~7 channels suffer from too low a 
rate, due to the small Z — > £ + £~ branching ratio. This leaves only the diphoton 6677 and 
dimuon bbfi + fi~ decay combinations. 

For all our calculations we assume an integrated luminosity of 600 fb" 1 for the LHC, and 
6000 fb" 1 [19] for the SLHC. For the VLHC, we consider both 600 fb" 1 and 1200 fb" 1 [23]. 
We choose a s {Mz) = 0.1185 [32], calculate signal and background cross sections using 
CTEQ5L [33] parton distribution functions, and our scale choice for all background processes 
is fip — — Vs- We include minimal detector effects by Gaussian smearing of the parton 



momenta according to ATLAS expectations [4], and take into account energy loss in the 
6-jets via a parameterized function. We assume a 6-tagging efficiency of e& = 50% for 
all hadron colliders. In addition, we include an efficiency of 79% [34] for capturing the 
H — > bb decay of the signal in its 40 GeV mass bin. We calculate all background processes 
using madgraph [35] except where otherwise noted, and retain a finite 6(c)-quark mass of 
4.6(1.7) GeV where relevant. Other detector efficiencies are given in the subsections relevant 
to the respective channels. 

A. The 6677 decay channel 

We perform the signal calculation, gg — > HH — > 6677, as in Refs. [20,22], includ- 
ing the effects of next-to-leading order (NLO) QCD corrections via a multiplicative factor 
K = 1.65(1.35) at LHC(VLHC) energies [36], using factorization and renormalization scales 
choices of m#. There is little scale variation left at NLO. We use exact matrix elements to 
incorporate the H — > bb and H — > 77 decays. 

The basic kinematic acceptance cuts for events at the (S)LHC and VLHC are: 

p T (b) > 45 GeV , \rj(b) | < 2.5 , AR(b, b) > 0.4 , 

m H - 20 GeV < m bi < m H + 20 GeV , 
pr( 7 ) > 20 GeV , (77(7) | < 2.5 , Ai2( 7>7 ) > 0.4 , 
m H - 2.3 GeV < m 77 < m H + 2.3 GeV , 

Ai?( 7 ,6)>0.4, (3) 

which are motivated first by requirements that the events can pass the ATLAS and CMS 
triggers with high efficiency [4,5], and that the 6-quark and photon pairs reconstruct to 
windows around the known Higgs boson mass, adjusted for an expected capture efficiency 
of 79% each [34]. We take the identification efficiency for each photon to be 80% at all 
machines considered [34]. 

As in the 4W signal case [20], we will later try to determine the Higgs boson self- 
coupling from the shape of the invariant mass of the final state. For that reason we do 
not apply any cuts which make use of the fact that the signal involves two heavy massive 
particles produced in a fairly narrow range of the btryy invariant mass. The only irreducible 
background processes are QCD bbyy, H(— > 77 )66 and H(—> 66) 77 production. However, 
there are multiple QCD reducible backgrounds resulting from jets faking either 6-jets or 
photons: 

• cc 77 - one or two fake b jets; 

■ bbj'y - one fake photon; 

• ccj 7 - one or two fake 6-jets, one fake photon; 

• jj 77 - one or two fake 6-jets; 

■ bbjj - two fake photons; 
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Table I. Expected photon and muon identification efficiencies, and misidentification probabil- 
ities for charm quarks and light jets as 6-quarks [4,5,19,37] and photons [4,5,38,39], at various 
hadron colliders. 









Pc^b 


Pj^b 


phi 


plo 


LHC 


80% 


90% 


1/13 


1/140 


1/1600 


1/2500 


SLHC 


80% 


90% 


1/13 


1/23 


1/1600 


1/2500 


VLHC 


80% 


90% 


1/13 


1/140 


1/1600 


1/2500 



• ccjj - one or two fake 6-jets, two fake photons; 

• 2331 ' one or two fake 6-jets, one fake photon; 

• jjjj - one or two fake 6-jets, two fake photons; 

• Hjj - one or two fake 6-jets, or two fake photons; 
■ Hj'y - one fake photon. 

Misidentified charm quarks must be considered separately from non-heavy flavor jets because 
of the grossly different rejection factors. Table I summarizes the expected rejection factors 
for charm and light jets to be misidentified as 6-jets and photons, as well as the expected 
photon and muon identification efficiencies. The probability to misidentify a light jet as a 
6-jet is significantly higher at the SLHC due to the high-luminosity environment [19]. The 
value quoted in Table I for Pj->b at the LHC is likely to be conservative; recent studies [37] 
using three dimensional 6-tagging have found a light jet rejection factor about a factor two 
better. Expectations for the probability to misidentify a light jet as a photon at the LHC 
vary considerably [4,5,38,39], so we perform two analyses, one conservative and the other 
optimistic, to cover this range. Since their design luminosities are similar, it is reasonable 
to assume that the rejection factors for light quarks and charm quarks, and the jet-photon 
misidentification probabilities, are similar for the LHC and the VLHC. Studies of how the 
high luminosity environment of the SLHC affects P c ^b and Pj^ have not yet been per- 
formed. In lieu of better estimates we therefore use the same values as for the LHC and 
VLHC. It should be noted that the rejection factors listed in Table I depend on the trans- 
verse momentum of the charm quark, Pt(c), or jet, pr(j)- The values listed in the Table 
correspond to the rejection factor in the px range which provides the largest contribution to 
the cross section. 

Except for the 6677 and bbjj backgrounds, all reducible backgrounds depend on whether 
one requires one or both 6-quarks to be tagged. Requiring only one tagged 6-quark results in 
a signal cross section which is a factor (2/e 6 — 1) = 3 larger than the one with both 6-quarks 
tagged. This larger signal rate comes at the expense of a significantly increased reducible 
background. As we shall demonstrate, the small gg — > HH — > 6677 cross section forces us 
to require a single 6-tag at the LHC in order to have an observable signal. At the SLHC, on 
the other hand, the much higher probability to misidentify a light jet as a 6-jet translates 
into an increase of the background which more than compensates the signal gain from using 
only a single 6-tag. In the following we therefore require a double 6-tag at the SLHC. For 
the VLHC we consider both single and double 6-tagging. 
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Figure 2. Distributions of the minimum lego plot (pseudorapidity - transverse plane) separation 
between (a) 6-jets and photons, and (b) the photons, for a SM signal of m# = 120 GeV and the 
QCD 6677 background; using the cuts of Eq. (3) but no minimum 6 — 7 separation. We include 
the NLO K-factor for the signal and a factor 1.3 for the QCD background. 

For a single 6-tag strategy, there is an additional combinatorical background when extra 
jets are present in the event. To estimate this background, one needs to interface the 
gg — > HH matrix elements with an event generator. Insight may also be gained from 
performing a calculation of HHj production, which presently does not exist. Since we 
calculate the signal cross section with cuts only at lowest order, we do not include the 
combinatorical background in our background estimate. 

At the level of cuts in Eq. (3), we observe two angular correlations which differ strongly 
between signal and background. The minimum separation between 6-jets and photons is 
typically much smaller for the QCD backgrounds as compared to the signal. The shape of 
the signal distribution reflects the fact that the bb and 77 pairs originate from decays of 
heavy scalar particles which recoil against each other in the transverse plane. The peak 
in the background Ai?(7, b) m i n distribution at small values is clearly due to the collinear 
enhancement from photon radiation off a 6-quark. The minimum separation between the 
photons, on the other hand, is smaller for the signal. We show the minimum photon-6 
and the photon-photon separation distributions in Fig. 2, for the HH signal and the 6677 
background at the LHC; all other background processes exhibit distributions qualitatively 
similar to those for QCD 6677 production. Based on these observations, we impose two 
additional angular cuts on the final state, which reduce the backgrounds by about an order 
of magnitude, but affect the signal at only the 15 — 20% level for ran = 120 GeV, and closer 
to 30% for m H = 140 GeV: 



Ai2(7,6) > 1.0 , Ai?( 7 , 7 ) < 2.0 . 



(4) 



Looking at Fig. 2, these to not appear to be the optimum values. However, the cuts are 
correlated, and we chose these values to roughly optimize S/ B while retaining a significant 
fraction of the signal. 

Tables II and III display the signal and QCD background cross sections for the (S)LHC 
and VLHC, including the signal K-factor, at the level of cuts in Eq. (3), adding Eq. (4), 
and finally with all efficiencies and misidentification probabilities applied, for both the con- 



servative ("hi". 



Pj- 



1/1600) and optimistic ("lo", Pj 



1/2500) assumptions. The 
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Table II. Expected cross sections [fb] (first three rows) for the mu = 120 GeV HH — ► 6677 
signal and QCD backgrounds, including the signal K-factors, at the (S)LHC. The background 
cross sections are scaled by a factor 1.3, as explained in the text. The QCD backgrounds cannot 
be calculated without cuts due to soft and collinear singularities. Each of the next four pairs of 
rows shows the cross sections including all detector efficiencies and fake tag rejection probabilities 
as described in the text, and the number of events expected, for each machine and background 
analysis. We assume an integrated luminosity of 600 fb^ 1 (6000 fb -1 ) for the LHC (SLHC). The 
Hjj, Hbb, ^77 and Hj'y backgrounds are discussed in the text and therefore not shown. 



analysis stage 


HH 


6677 


CC77 


bbjj 


cc7j 


jjn 


bbjj 


ccjj 


ijjj 


jjjj 


E(bkg) 


before cuts 
+ Eq. (3) 
+ Eq. (4) 


0.15 
0.043 
0.035 


0.056 
0.0060 


0.42 
0.0215 


65 
8.28 


250 
17.0 


11 

0.84 


2.5xl0 4 
4520 


2.5xl0 4 
4520 


7700 
364 


5xl0 6 
4xl0 5 


5xl0 6 
4xl0 5 


N LHC (hi) 


0.0106 
6 


0.0029 
2 


0.0020 
1 


0.0031 
2 


0.0013 
1 


0.0077 
5 


0.0013 
1 


0.0003 



0.0030 
2 


0.0022 
1 


0.0233 
14 


X fP l L°HC 

N LHC (lo) 


0.0106 
6 


0.0029 
2 


0.0020 
1 


0.0020 
1 


0.0008 



0.0077 
5 


0.0005 



0.0001 



0.0017 
1 


0.0009 
1 


0.0186 
11 


x e SLHC 

NsLHc(hi) 


0.0035 
21 


0.0010 
6 


0.0001 



0.0010 
6 


0.0001 



0.0010 
6 


0.0004 
3 


0.0000 



0.0003 
2 


0.0003 
2 


0.0042 
25 


X e ' P l SLHC 

Nslhc{Io) 


0.0035 
21 


0.0010 
6 


0.0001 



0.0007 

4 


0.0000 



0.0010 
6 


0.0002 
1 


0.0000 



0.0002 
1 


0.0001 
1 


0.0033 
20 



QCD background normalization uncertainty is rather large at LO, and unfortunately none 
of these processes is known at NLO. To estimate the effect of a possible NLO increase of 
the background rates, we scale each of the background cross sections by a factor 1.3. Note 
that we are not making any statement about unknown higher order corrections. Instead, 
we attempt to be conservative and show that our results do not critically depend on the 
background normalization. 

Before final state identification, the O(a^) jjjj background dominates over all others by 
two orders of magnitude. The angular cuts of Eq. (4) do improve the signal to background 
ratio by an order of magnitude, but it is the cumulative effect of large rejection factors for 
misidentifying light jets as photons or fo-jets that brings the QCD backgrounds down to a 
manageable level. 

The single Higgs- resonance backgrounds are for the most part negligible, so we do not 
include them in Tables II and III. The Hjj cross section is approximately a factor 6 — 20 
smaller than the signal [40] ; the Hbb cross section is a factor 20 — 60 smaller. Although no 
calculations of if 77 and Hj'y production exist yet, one expects that these backgrounds are 
also negligible. All subsequent numerical results include the Hjj background, whereas we 
neglect the Hbb, if 77 and Hj'y backgrounds. 

Summing all background cross sections we find that S/B ~ 1/1 is possible at the SLHC, 
and we anticipate a still respectable S/B ~ 1/2 at the LHC. At the VLHC, with one tagged 
6-quark, we obtain a signal to background ratio of about 1/1, while a double 6-tag yields 
to S/B ~ 2 — 3.5. Of course, even small changes in expected fake-6 rejection factors could 
change how the analysis would be optimized. Our results are meant only to highlight the 



8 



Table III. Expected cross sections [fb] (first three rows) for the m# = 120 GeV HH — > 6677 
signal and QCD backgrounds, including the signal K-factors, at the VLHC. The background cross 
sections are scaled by a factor 1.3, as explained in the text. The QCD backgrounds cannot be 
calculated without cuts due to soft and collinear singularities. Each of the next pairs of rows shows 
the cross sections including all detector efficiencies and fake tag rejection probabilities as described 
in the text, and the number of events expected for an integrated luminosity of 600 fb^ 1 , for each 
of the two background analyses. We show results for both single and double 6-tagging. The Hjj, 
Hbb, H'yy and H31 backgrounds are discussed in the text and therefore not shown. 



analysis stage 


HH 


6677 


CC77 


bbjj 


ccryj 


jjn 


bbjj 


ccjj 


1333 


jjjj 


E(bkg) 


before cuts 
+ Eq. (3) 
+ Eq. (4) 


15.9 
3.12 
2.70 


1.8 
0.14 


23 
1.23 


3600 
417 


14000 
1020 


280 
25.0 


1.6xl0 6 
4.2xl0 5 


1.6xl0 6 
4.2xl0 5 


2.3xl0 5 
13300 


2.7xl0 8 
3.0xl0 7 


2.7xl0 8 
3.0xl0 7 


1 6-tag 


x e r VLHC 

N(hi) 


0.810 

486 


0.067 
40 


0.116 
70 


0.156 
94 


0.075 
45 


0.228 
137 


0.122 
73 


0.024 
14 


0.095 
57 


0.164 

98 


1.048 
629 


x e • Pvlhc 
N(lo) 


0.810 

486 


0.067 
40 


0.116 
70 


0.100 
60 


0.048 
29 


0.228 
137 


0.050 
30 


0.010 
6 


0.061 
36 


0.067 
40 


0.747 

448 


2 6-tags 


x e r VLHC 

N(hi) 


0.270 
162 


0.022 
13 


0.005 
3 


0.052 
31 


0.003 
2 


0.001 



0.041 
25 


0.001 
1 


0.000 



0.001 
1 


0.126 
76 


x e " Pvlhc 
N(lo) 


0.270 
162 


0.022 
13 


0.005 
3 


0.033 
20 


0.002 
1 


0.001 



0.017 
10 


0.000 



0.000 



0.000 



0.080 
47 



potential capability of such a search. 

Our estimates also reveal that the range of fake photon rejection probabilities is not 
so significant. The largest background in most cases is jj'77, where the photons are real 
but one or two 6-tags are falsely identified — at the SLHC the double 6-tag requirement 
brings this background to the same level as the real 6677 component. The irreducible 6677 
background in all cases constitutes only a small fraction of the total background. 

As shown in Table III, requiring two 5-tags instead of one at the VLHC reduces the 
overall background by a factor 8-9, but the signal by only a factor 3. As a result, both 
cases yield similar sensitivity bounds for the Higgs self-coupling A. However, we note that 
the higher event rate with one 6-tag will provide better control of experimental systematic 
uncertainties, so this may be the preferred strategy. 

In addition to the backgrounds considered so far, 6677 events (or their fakes) may also be 
produced in double parton scattering (DPS), or from multiple interactions occurring from 
separate pp collisions in the same bunch crossing at high-luminosity running. In principle, 
one can identify multiple interactions by a total visible energy measurement or by tracing 
some final particle tracks back to distinct event vertices, but this may not always be possible 
in practice. For example, for 6677 events where the photon and bb pairs occur in different 
interactions, the latter method relies solely on tracks of particles associated with the hadronic 
activity accompanying the photon pair. If these particles are soft, the two vertices may not 
be clearly resolvable. 
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To estimate the cross sections from DPS and multiple interactions, we use the approxi- 
mation outlined in Ref. [41]. In both cases, the dominant contribution arises from multi-jet 
production where several jets are misidentified as 6-quarks or photons. After applying the 
cuts listed in Eqs. (3) and (4), the DPS and multiple interaction backgrounds are still several 
times larger than the signal. However, to discriminate them from regular single interaction 
events, one can exploit the independence and pairwise momentum balance of the two scat- 
terings in DPS or multiple interaction events, similar to the strategy employed in the DPS 
analysis carried out by the CDF collaboration [42]. Rejecting events where two sets of 
transverse momenta independently add up to a value close to zero will obviously strongly 
suppress the DPS and multiple interaction background. The signal, on the other hand, is 
only minimally affected by such a cut. Requiring that events which pass the cuts listed in 
Eqs. (3) and (4) do not satisfy either 

|p T (6) + pr(7i)| < 20 GeV and |p T (6) + pr(7 2 )| < 20 GeV (5) 

or 

|pr(6) + Pr(7i)| < 20 GeV and \p T (b) + pV(7 2 )| < 20 GeV (6) 

totally eliminates the DPS and multiple scattering backgrounds (within the limits of our 
ability to simulate detector effects), but reduces the signal cross section by about 7%. This 
has essentially no influence on the Higgs self-coupling sensitivity bounds. 

Extracting the Higgs boson self-coupling follows the same path as for the AW final state 
used for larger Higgs masses [20]. To discriminate between signal and background, we use 
the visible invariant mass, m v i S , which for this final state is the invariant mass of the Higgs 
boson pair, corrected for energy loss of the 6-jets. We show this in Fig. 3 for rnn = 120 GeV 
at the LHC, and in Figs. 4 and 5 for m H = 120 GeV and m H = 140 GeV at the SLHC 
and VLHC. We do not show the uih = 140 GeV case for the LHC, since we expect only 
about two signal events for an integrated luminosity of 600 fb _1 . Figs. 3-5 show that the 
background distribution peaks close to the threshold, whereas the signal distribution reaches 
its maximum at a somewhat higher value. This is due to the destructive interference between 
the triangle and box diagrams contributing to gg — > HH . It is responsible for an increase 
in the signal cross section and a shift in the m vis peak position towards lower values, if 
we assume A < Xsm, and vice versa. The shape of the visible invariant mass distribution 
thus helps to discriminate signal and background and to probe the Higgs self-coupling, A. 
Increasing inn from 120 GeV to 140 GeV reduces the signal (background) cross section by 
about a factor 3 (2). 

To derive quantitative sensitivity bounds on A we perform a x 2 test of the m v ; s distribu- 
tion, similar to that described in Ref. [20]. Except for the Higgs self-coupling, we assume 
the SM to be valid. As in all previous analyses, we multiply the LO differential cross sec- 
tions of the QCD background processes by a factor 1.3. As mentioned before, this is not a 
guess of the higher order corrections, which must either be computed, or the rates measured 
sufficiently precisely. However, this way we ensure that our results do not critically depend 
on the absolute normalization of the background rates, while of course they will depend on 
the uncertainty associated with the determination of the background rate: we allow for a 
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Figure 3. The visible invariant mass distribution, m v i s , in pp — ► 6677, after all kinematic cuts 
(Eqs. (3) and (4)), for the conservative (short dashed) and optimistic (long dashed) QCD back- 
grounds and a SM signal of m# = 120 GeV (solid) at the LHC. The dotted and short dash-dotted 
lines show the signal cross section for Xhhh = X/Xsm = and 2, respectively. To illustrate how 
the reducible backgrounds dominate the analysis, we also show the irreducible QCD 6677 back- 
ground by itself (long dash-dotted). We include the NLO K-factor for the signal and a factor 1.3 
for the QCD backgrounds. 

normalization uncertainty of 10% for the SM signal plus background rate. We express limits 
on the deviation of the Higgs self-coupling from the SM value in terms of AXhhh, where 

AXhhh = Xhhh — 1 = t 1 • (7) 

XsM 

We summarize our results in Table IV. The bounds obtained using the conservative 
background estimate (labeled "hi") are 10 — 20% less stringent than those found using the 
more optimistic scenario (labeled "lo"). At the SLHC, for m H = 120 GeV, a vanishing Higgs 
self-coupling can be ruled out at the 90% CL. Limits for m H = 140 GeV are a factor 1.2-2 
weaker than those for m H = 120 GeV. 

It may be possible to subtract large parts of the reducible backgrounds which do not 
involve charm quarks using the following technique. Due to the their large cross sections 
(see Tables II and III), one can fairly accurately determine the m vis distributions of the 
individual processes, Hjj, bfryj, bbjj, jju, 1333 an d 3333 production, imposing the same 
cuts as in the HH — > 6677 analysis (Eqs. (3) and (4)). If the photon-jet and light jet— 6 
misidentification probabilities are independently measured in other processes such as prompt 
photon [43] and W+ jets production, one can simply subtract these backgrounds. For the 
background processes involving charm quarks, on the other hand, this procedure will be 
more difficult to realize, since the smaller charm quark mass and the shorter charm lifetime 
result in a charm quark tagging efficiency much lower than that for 6-quarks. The columns 
labeled "bgd. sub." list the limits achievable if the non-charm reducible contributions to the 
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Figure 4. The visible invariant mass distribution, m v i s , in pp — > 6677, after all kinematic cuts 
(Eqs. (3) and (4)), for the conservative (short dashed) and optimistic (long dashed) QCD back- 
grounds and SM signals of uih = 120 (upper) and 140 GeV (lower) at the SLHC. The dotted and 
short dash-dotted lines show the signal cross section for Xhhh = X/ Xsm = and 2, respectively. 
To illustrate how the reducible backgrounds dominate the analysis, we also show the irreducible 
QCD 6677 background by itself (long dash-dotted). We include the NLO K-factor for the signal 
and a factor 1.3 for the QCD backgrounds. 

background were subtracted with 100% efficiency, but none of the charm quark backgrounds 
could be reduced. Our results show that reducing the background beyond what can be 
achieved with kinematic cuts may considerably improve the bounds on Xhhh at the LHC 
and SLHC, where the HH — > 6677 process is statistics limited. The bounds achievable at 
the SLHC (VLHC) by analyzing 6677 production are a factor 2.5 - 6 (2 - 3) more stringent 
than those from the bbr + T~ channel [22]. 

Due to the small number of events, the LHC and SLHC sensitivity limits depend signifi- 
cantly on the SM cross section normalization uncertainty. For example, for a normalization 
uncertainty of 30% on the SM signal plus background rate, the achievable bounds on X HH h 
are almost a factor 2 weaker than those obtained for a normalization uncertainty of 10%. 
This SM cross section normalization uncertainty depends critically on knowledge of the QCD 
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Figure 5. The visible invariant mass distribution, m v i s , in pp — > 6677, after all kinematic 
cuts (Eqs. (3) and (4)), for the conservative (short dashed) and optimistic (long dashed) QCD 
backgrounds and SM signals of mu = 120 (upper) and 140 GeV (lower) at a VLHC. The dotted and 
short dash-dotted lines show the signal cross section for Xrhh = A/ Xsm = and 2, respectively. 
To illustrate how the reducible backgrounds dominate the analysis, we also show the irreducible 
QCD 6677 background by itself (long dash-dotted). We include the NLO K-factor for the signal 
and a factor 1.3 for the QCD backgrounds. 

corrections to the signal and the ability to determine the background normalization. The 
NLO QCD corrections to gg — > HH are currently known only in the infinite top quark mass 
limit [36]. To ensure the 10% required precision on differential cross sections we would need 
the NLO rates for finite top quark masses, as well as the NNLO corrections in the heavy 
top quark mass limit. For the background normalization one can rely on either calcula- 
tions of the QCD corrections or data. As mentioned before, none of these NLO background 
calculations are available. Since there are many processes contributing to the background, 
and most of them involve hundreds of Feynman diagrams already at tree level, NLO cal- 
culations appear feasible only if automated one-loop QCD tools become available in the 
next few years. In the absence of such NLO results, one may be able to fix the background 
normalization instead by relaxing the bb and 77 invariant mass cuts of Eq. (3) and/or the 
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Table IV. Expected Higgs self-coupling 68.3% CL (la) sensitivity limits, expressed as 

AXhhh = 1> f° r the var i° us hadron collider options and background analyses presented in 

the text. There are not enough events at the LHC for ran = 140 GeV to perform a measurement 
of A. The LHC and VLHC analyses employ a single 6-tag strategy, while the high- luminosity 
conditions at the SLHC force a double fe-tag requirement. 







m H = 120 GeV 






m H = 140 GeV 




machine 


"hi" 


"lo" 


bkg. sub. 


"hi" 


"lo" 


bkg. sub. 


LHC, 600 fb- 1 


+1.9 
-1.1 


+1.6 
-1.1 


+0.94 
-0.74 








SLHC, 6000 fb" 1 


+0.82 
-0.66 


+0.74 
-0.62 


+0.52 
-0.46 


+ 1.7 
-0.9 


+1.4 
-0.8 


+0.76 
-0.58 


VLHC, 600 fb" 1 


+0.44 
-0.42 


+0.42 
-0.40 


+0.32 
-0.30 


+0.82 
-0.62 


+0.66 
-0.54 


+0.38 
-0.34 


VLHC, 1200 fb" 1 


+0.32 
-0.30 


+0.30 
-0.28 


+0.26 
-0.22 


+0.76 
-0.58 


+0.62 
-0.50 


+0.36 
-0.32 



cuts of Eq. (4) and extrapolating from regions in m b i, m 77 , AR(j, b) m i n and AR(j, 7) where 
the background dominates, back into the analysis region. This technique should make it 
possible to determine the background normalization to about 10% at the LHC and SLHC, 
and to about 2% at the VLHC. Both methods rely on Monte Carlo simulation to correctly 
predict the m vis distribution shape. 

The bounds listed in Table IV should be compared with those achievable at e + e~ linear 
colliders. A linear collider with ^/s = 500 GeV and an integrated luminosity of 1 ab _1 can 
determine A with a precision of about 20% in e+e - — > ZHH for m H = 120 GeV [18]. For 
m H > 120 GeV, the H — > bb branching ratio and the e + e~ — > ZHH cross section both 
fall off quickly. Since the background cross section decreases only slightly, S/B, and thus 
the bounds on A obtainable from e + e~ — > ZHH, worsen rapidly with increasing values of 
m H . By m H = 140 GeV they are at only the 50% level [22]. From Table IV it is clear 
that the LHC will be able to provide only a first rough measurement of the Higgs self- 
coupling for m H = 120 GeV. A luminosity-upgraded LHC will be able to make a more 
precise measurement. However, the sensitivity bounds on A obtained from 6677 production 
for itlh = 120 GeV (m# = 140 GeV) will be a factor 2-4 (1.2 - 3) weaker than those 
achievable at a linear collider. In contrast, the sensitivity at a VLHC will approach this 
level of precision. It should be noted that if the SM cross section normalization uncertainty 
could be reduced to a few percent, a VLHC could reach precision similar to that foreseen 
for CLIC [17] (e + e- collisions at 3 TeV center-of-mass energy). 

B. The decay channel 

The bbfi + fi~ signal calculation proceeds as in the 6677 case. The basic kinematic accep- 
tance cuts for events at the LHC and VLHC are: 

pr(b) > 45 GeV , \rj(b)\ < 2.5 , AR(b, b) > 0.4 , 
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m H - 20 GeV < m bh < m H + 20 GeV , 
p T (pi) > 15 GeV , \rj(pi)\ < 2.4 , AR^fi) > 0.4 , 
m H — 5 GeV < < m H + 5 GeV , 

AR(b, fi) > 0.4 , (8) 

where again the muon invariant mass window is chosen to accept 79% of the H — » fi + fi~ 
decay after detector effects. The signal cross section at the LHC (VLHC) for tuh = 120 GeV 
before taking into account any efficiencies is 2.4 ab (0.21 fb), approximately one order of 
magnitude smaller than the 6677 channel. For larger Higgs boson masses the ratio is even 
smaller, due to the H — > branching ratio, which decreases much more rapidly with m H 

than that for H — > 77 (see Fig. 1). Once efficiencies are taken into account, we expect less 
than one signal event at the LHC. The SLHC would see 2 - 3 signal events for m H = 120 GeV 
if one assumes that both 6-quarks are tagged, too few for a meaningful coupling extraction. 
At a VLHC there would be about 60 signal events for an integrated luminosity of 600 fb -1 , 
single 6-tag requirement, and the same value of We therefore concentrate on the VLHC 
in the following, and require only one 6-tag. 

A potential advantage of the bbfi + fi~ final state is the smaller number of processes con- 
tributing to the background. The main contributions to the background originate from QCD 
bbn + n~, ccfi + fi~ and jjfi + fi~ production, where the fi + fi~ pair originates from an off-shell 
Z-boson or photon. In the latter two processes, either a charm quark or light jet is misiden- 
tified as a 6-quark. We calculate the background processes at LO using MCFM [44] and find 
that their sum is more than a factor 200 larger than the signal. The signal to background 
ratio improves by a factor 5 if we additionally require 

AR(fi, n) < 2 , (9) 

whereas the signal cross section falls by only about 20%. The Hjj background is negligible 
compared with jj{i + fi~. The final signal to background ratio of S/B « 1/50 contrasts 
starkly with the S/B ~ 1/1 ratio the 6677 channel enjoys. If instead both fo-jets are tagged, 
the signal to background ratio improves by an additional factor 2. However, the signal 
cross section is reduced by a factor 3, which yields sensitivity bounds for Xhhh which are 
somewhat weaker than those obtained from single 6-tag data. 

Shrinking the invariant mass window could also reduce the background. The value 

in Eq. (8) was chosen assuming ATLAS detector muon momentum resolution [4]. The CMS 
detector [5] likely can use a smaller window, |m# — m M/1 | < 3 GeV, which would reduce the 
background by approximately a factor 1.7. 

The small signal cross section combined with the very large background make it essen- 
tially impossible to determine the Higgs boson self-coupling in pp — > bbfi + fi~. We quantify 
this by performing a x 2 test on the m vis distribution, similar to that described in Sec. Ill A. 
Since the signal cross section is too small to be observable at the LHC and SLHC, we de- 
rive bounds only for a VLHC. As before, we include the effects of NLO QCD corrections 
via multiplicative factors: K = 1.35 for the signal [36], K = 0.81 for bbfj, + fi~ and ccfi + fi~ 
production, and K = 0.91 for the jjfi + fi~ background [44]. Allowing for a normalization 
uncertainty of 10% of the SM cross sections, for uih = 120 GeV we find la bounds of 

-3.0 < A\ HHH < 4.2 (10) 
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at the VLHC for an integrated luminosity of 600 fb -1 . If the jjfi + fi~ background can be 
subtracted as described in Sec. Ill A, the limits improve by about a factor 1.4. Using the 
CMS dimuon mass window instead, the bound improves by about a factor 1.3. Nevertheless, 
this is about an order or magnitude weaker than the limits from HH — > 6677. 

IV. SUPERSYMMETRIC HIGGS BOSONS 

The MSSM requires two Higgs doublets, in contrast to one in the SM, to give mass to the 
up-type and the down-type fermions and to avoid anomalies induced by the super symmetric 
fermionic partners of the Higgs bosons. This results in the presence of five physical Higgs 
bosons: a charged pair H ± 1 two neutral scalars h° and H°, and a pseudoscalar A . The two 
scalars are mixed mass eigenstates, the lighter always having a mass m h < 135 GeV [45]. 
At leading order, the entire MSSM Higgs sector is described by two parameters, usually 
taken to be the ratio of the two Higgs doublets' vacuum expectation values, tan/3, and 
the pseudoscalar Higgs mass, uia- In the region itla ~ 150 GeV, all heavy Higgs bosons 
A, H, H ± have similar masses, much larger than the light scalar Higgs mass. In this so- 
called decoupling regime the light Higgs boson h strongly resembles a SM Higgs boson of 
the same mass. It will be difficult to distinguish between the SM and the MSSM Higgs 
sectors through measurements of its properties [3,7]. 

Assuming bottom-tau mass unification, only two regions of tan/5 are allowed: either 
small values, tan/3 < 3, or large values, tan/3 ^ 30. Direct searches for the heavy Higgs 
bosons are particularly promising in the large tan/3 regime, since in the decoupling limit 
the bottom Yukawa coupling to heavy Higgses is m;,tan/3. As a result, 6-quark initiated 
processes, such as bb — > H, may have cross sections enhanced by up to three orders of 
magnitude over the corresponding SM rates for sufficiently large values of tan /3. In contrast, 
for small values of tan/3 these direct searches fail, because the dominant Yukawa coupling 
becomes m t / tan (3 3> m& tan (3. 

At the LHC, associated production of two neutral MSSM Higgs bosons via gluon fusion 
occurs for all six possible combinations [29]. In principle, these processes probe the various 
Higgs boson self-couplings, \ijk- However, for large tan/3 the continuum box diagrams 
are enhanced by the Yukawa coupling squared, while the triangle loop diagram with an 
intermediate Higgs boson is enhanced by only one power of the large Yukawa coupling: for 
large tan (3 the resonance diagrams are suppressed by 1/ tan (3 as compared to the continuum 
production diagrams. For tan/3 = 50 we find that the effect of vanishing self couplings 
\ij k = is at maximum at the percentage level. 

For tan/3 ^ 30 and itla ~ 150 GeV, MSSM Higgs pair production cross sections can be 
sizable, reaching values up to 100 fb, compared to a few tens of fb in the SM. The largest 
cross sections occur for two heavy states AH, AA, HH and large values of tan (3, due to 
the enhanced coupling of these states to 6-quarks. In this regime the most promising final 
state is bbfi + fi~ since the ratio of the muon and the bottom Yukawa couplings is preserved 
in the MSSM, but the branching ratio to photons is highly suppressed, typically by several 
orders of magnitude compared to the SM Higgs boson of equal mass. Unfortunately, a main 
background for this is MSSM bbH/A,H/A — > production [46]. Whether the Higgs 

pair signal could be extracted out of this would require a more detailed investigation which 



16 



BR 



10 



-2 

10 



-3 

10 



150 200 250 300 350 400 450 500 

m A [GeV] 

Figure 6. Lowest order cross section and branching fractions for pair production of light MSSM 
scalar Higgs bosons, pp — > hh, with subsequent decay hh — > ^677, as a function of the pseu- 
doscalar Higgs mass txia- We fix tan/3 = 3, set the squark mass parameters to 1 TeV, and assume 
maximal mixing with A t = 2.5 TeV [47]. We do not take into account supersymmetric decay 
modes of the heavy Higgs boson H [48]. The light Higgs boson mass is above the LEP limit of 
ran > 114.4 GeV [49] for m A > 190 GeV. No cuts or detection efficiencies are included. The 
dashed horizontal line shows the lowest order SM gg — > HH cross section for run = 120 GeV. 

we do not find likely to be fruitful. 

In the small tan j3 regime it is much more difficult to distinguish the SM and the MSSM 
Higgs sectors. None of the heavy Higgs bosons will be directly observable at the LHC for 
tan/3 P$ 20, if we rely on the usual decays to fermions. We find that, for small values 
of tan /3, gg — > H — > hh offers the best chance to detect the heavy scalar Higgs boson, 
H: for tan/3 < 5 the H — > hh branching ratio is sizable [50]. To take into account off- 
shell effects we compute the full pp — > hh production rate. As in the SM, we expect the 
6677 final state to be most promising in the decoupling regime, with increased rate due 
to the intermediate H resonance. We show the h — > bb and h — > 77 branching fractions 
and lowest order gg — > hh — > 6677 cross section as a function of in Fig. 6. The light 
Higgs boson mass increases from m h = 108 GeV for m A = 150 GeV to a plateau value of 
rrih = 122 GeV in the large tua limit. A few structures in the cross section plot require 
further explanation. First, the heavy scalar Higgs mass crosses the threshold itlh > 2mh 
around itla ~ 225 GeV, which enhances the hh cross section by almost a factor 100. Second, 
the kink at ~ m H = 350 GeV represents the top threshold in the top triangle loop. At 
the same time we see the onset of the H — > tt decay channel, which for larger values of m A 
dominates over H — > hh, so the cross section decreases rapidly. Nevertheless, the MSSM 
signal rate is still enhanced over the SM rate <JsM(bkyy) w 0.09 fb for values of m A as large 
as 500 GeV. 

Unfortunately, the angular cuts of Eq. (4) which are needed to suppress the background, 
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Figure 7. The visible invariant mass distribution, m v i s , for MSSM light scalar Higgs pair pro- 
duction at the LHC, pp — > hh — > 6677, for tan/3 = 3. The light Higgs mass for tua = 325 GeV is 
120.8 GeV and for tua = 400 GeV it is 122.2 GeV. For comparison, we also show the distribution 
for SM Higgs pair production (ma = 120 GeV). 

together with the standard 6677 identification cuts of Eq. (3), force the differential cross 
section to vanish for m vis < 250 GeV. Pair production of light supersymmetric Higgs bosons 
will thus be unobservable for m A < 280 GeV. When taking into account detection efficiencies, 
we find that hh production at the LHC should be observable at the 5a level for 320 < itla < 
375 GeV (310 < m A < 425 GeV) for an integrated luminosity of 300 fb^ 1 (600 fb^ 1 ) and 
tan/3 = 3. The signal would be rather spectacular: due to s-channel H exchange, the 
differential cross section peaks for m vis m H) as shown in Fig 7. Compared to the SM case 
the cross section is enhanced by more than an order of magnitude in the resonance region, 
where it depends on the Hhh and Hff couplings. Since MSSM heavy scalar H production 
with decay into fermions is unobservable at the LHC in the small tan f3 region, this implies 
that hh production can measure only a combination of Xuhh and the Hff couplings, but 
not the individual couplings. 




V. DISCUSSION AND CONCLUSIONS 

After discovery of an elementary Higgs boson and tests of its fermionic and gauge boson 
couplings, experimental evidence that the shape of the Higgs potential has the form required 
for electroweak symmetry breaking will complete the proof that fermion and weak boson 
masses are generated by spontaneous symmetry breaking. One must determine the Higgs 
self-coupling to probe the shape of the Higgs potential. 

Only Higgs boson pair production at colliders can accomplish this. Numerous stud- 
ies [15-18] have established that future e + e~ machines can measure A at the 20 — 50% level 
for ran < 140 GeV. Very recent studies [19-21] determined that the prospects at hadron 
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colliders for 150 < tuh < 200 GeV are similarly positive, but that the mu < 140 GeV region 
would be very difficult to access [22]. We have tried to rectify the situation in this paper 
by considering highly efficient, lower-background rare decay modes: 6677 and bbji + \i~ . The 
latter suffers from very low rate and considerable background from the Breit-Wigner tail of 
bbZ production, and does not appear to be useful. This is not surprising upon comparison 
to our bbr + T~ study [22]. 

However, the 6677 channel shows considerable promise. Imposing photon-photon and 
photon-5 separation cuts could result in a signal to background ratio of 0(1) or better. 
Since the irreducible QCD 6677 background is small compared to the reducible background 
originating from light jets or charm quarks mistagged as 6-quarks, or from jets misidenti- 
fied as photons, the signal to background ratio depends on the particle misidentification 
probabilities, and the required number of 6-tags. 

We find that the LHC, with an integrated luminosity of 600 fb _1 or more, could make a 
very rough first measurement for mu = 120 GeV (with ~ 6 signal events), but would not 
obtain useful limits for m H = 140 GeV at all due to the lack of signal events. It would 
require a luminosity-upgraded run (SLHC, 6000 fb" 1 ) to rule out A = at the 90% CL for 
m H = 120 GeV, and to make a 50 - 80% measurement at the la level. A 200 TeV VLHC, 
in contrast, would make possible a 20 — 40% measurement of A, competitive with future 
e + e~ collider capabilities. We note, however, that current understanding of hadron collider 
Higgs boson phenomenology doesn't provide for the necessary precision knowledge of Higgs 
branching ratios to complement this. It is likely that an e + e" collider would still be required 
to fill this role. Although a luminosity-upgraded LHC cannot compete with a linear collider 
for Higgs masses m# < 140 GeV, a Higgs self-coupling measurement at the SLHC will still 
be interesting if realized before a linear collider begins operation. 

To fully exploit future hadron collider potential to measure the Higgs self-coupling, we 
need an accurate prediction of the SM 6677 rate. It is mandatory that the residual theoretical 
cross section uncertainty be reduced to the 10 — 15% level for any HH analysis to be 
meaningful. We will need similar precision on background rates probably from experiment 
by extrapolating from background-dominated phase space regions to that of the signal. 

Probably the most exciting result of this analysis is the MSSM case: the heavy MSSM 
Higgs scalar can decay into two light Higgs bosons if tan/5 < 5. This region of parameter 
space poses a serious challenge to the LHC, because none of the usual heavy Higgs searches 
will detect a hint of the two Higgs doublets required in the MSSM. Resonant production 
of the heavy scalar Higgs in gluon fusion and its subsequent decay into light Higgs bosons, 
which then decay to 6677, has two effects on the cross section as compared to the SM case: 
the total rate is enhanced by about an order of magnitude and the hh invariant mass peaks 
at the heavy Higgs mass. Even though our analysis is not at all optimized for resonant 
MSSM production, we find a 5a discovery region for tan j3 = 3 and 310 < itla < 425 GeV at 
the LHC. Even though the discovery reach of this channel does not extend to much larger 
values of tan (3 it still ensures the observation of one heavy Higgs boson in a region preferred 
by bottom-tau unification, inaccessible by other MSSM Higgs searches. 
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